Ingest Custom Actions - AWS Data Firehose

Moesif can directly ingest your API logs or custom actions from an AWS firehose instance.

What are actions?

Actions are custom usage events that you directly send to Moesif. Actions have an action name (like “Signed Up” or “Finished Job”) which represents the raw event. You can also include arbitrary metadata with an action which enables you to create billable metrics, usage reporting, and more. For more info, see docs on actions.

How it works

The AWS Firehose integration works by using the AWS HTTP Endpoint for the Firehose destination. Then your applications can insert your custom actions into the firehose which will be sent to Moesif. For example, below shows an API Gateway which sends events to the firehose.

Diagram of Moesif AWS API Gateway logging architecture

Limitations

Enforcing quotas and governance rules will not work without a server integration. You can always install a server integration later if this capability is required.

How to install (actions)

Create a Kinesis Data Firehose

Click the below Launch Stack button to open CloudFormation which will automatically create a new Data Firehose configured with Moesif Actions API as the destination.

Launch CloudFormation Stack

Ensure you checkbox I acknowledge that AWS CloudFormation might create IAM resources as shown below. Then click Create Stack.

AWS API Gateway Cloudformation create stack

If you experience errors or want to create the Firehose manually, follow these instructions.

After the stack is created, search for “Firehose” in the AWS Console search bar and select Kinesis Data Firehose.

Insert an action

Now that your firehose is created, save an example action into your firehose. The firehose message must match the schema for an action which is available here.

You’ll want to set a few fields like below:

  • action_name is a string and should include name of the event such as “Processed Payment Transaction”.
  • company_id is the customer identifier (see companies).
  • transaction_id should be a random UUID for this event which Moesif uses for deduplication (docs on Moesif idempotency).
  • request.time represents the transaction time as an ISO formatted string.
  • metadata is an object which includes any custom properties for this event. By setting metadata, you can bill on arbitrary metrics, create metrics on them, etc. For example, if the action name is “Processed Payment Transaction”, you can include an amount and the currency to bill on the total amount.

For full schema and available fields, see Actions API Reference.

An example action is below:

{
  "action_name": "Processed Payment Transaction",
  "request": {
    "time": "2024-03-01T04:45:42.914"
  },
  "company_id": "12345",
  "transaction_id": "a3765025-46ec-45dd-bc83-b136c8d1d257",
  "metadata": {
    "amount": 24.6,
    "currency": "USD",
    "time_seconds": 66.3
  }
}

In the above example, the action is created whenever a payment is processed. There are also two metrics we are tracking as part of the action (the amount of the payment and how long the job took). You can create billable metrics and usage reports from these attributes.

Manual Firehose Creation

If you cannot use the CloudFormation template or having issues, you can manually create and configure a Kinesis Data Firehose using the HTTP destination.

Go to Amazon Kinesis within the AWS Console and select Create Delivery Stream.

  • For the source, select Direct PUT or other sources.
  • For the destination, select HTTP Endpoint and enter the following:
  • For HTTP endpoint URL, enter https://api.moesif.net/v1/partners/aws/kinesis/actions
  • For Access key, enter your Moesif Application Id.
  • For Buffer interval, enter 60 seconds.
  • Enable GZIP compression.

Updated: